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Method and apparatus for controlling a virtual reality 
graphics system using interactions 

5 Description 

The present invention generally relates to graphics 
systems for virtual reality (VR) applications and 
specifically relates to a method and an apparatus for 
10 controlling such a VR graphics system using 
interactions as claimed in the preambles of the 
respective independent claims. 

A VR graphics system which is concerned in this case is 
15 evident from DE 101 2 5 075 Al , for example, and is used 
to generate and display a multiplicity of three- 
dimensional views which together represent a virtual 
three-dimensional scene. In this case, such a scene is 
usually correspondingly visualized using stereoscopic 
20 projection onto a screen or the like. So-called 
immersive VR systems which form an intuitive man- 
machine (user) interface for the various areas of use 
(fig. 1) are already relatively widespread. Said 
graphics systems use a computer system to highly 
25 integrate the user into the visual simulation. This 
submersion of the user is referred to as "immersion" or 
an "immersive environment". 

As a result of the fact that three-dimensional data or 
3 0 objects are displayed to scale and as a result of the 
likewise three-dimensional ability to interact, these 
data or objects can be assessed and experienced far 
better than is possible with standard visualization and 
interaction techniques, for example with a 2D monitor 
3 5 and a correspondingly two-dimensional graphical user 
interface. A large number of physical real models and 
prototypes may thus be replaced with virtual prototypes 
in product development. A similar situation applies to 
planning tasks in the field of architecture, for 
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example. Function prototypes may also be evaluated in a 
considerably more realistic manner in immersive 
environments than is possible with the standard 
methods . 

5 

Such a VR simulation is controlled in a computer-aided 
manner using suitable input units (referred to below, 
for the purpose of generalization, as "interaction 
units" since their function goes beyond pure data 

10 input) which, in addition to pushbuttons, have a 
position sensor which can be used to likewise 
continuously measure the spatial position and 
orientation of the interaction unit in order to carry 
out the interactions with the data which are displayed 

15 in the form of a scene (scene data) . Such an 
interaction unit and a corresponding three-dimensional 
user interface are disclosed, for example, in DE 101 32 
243 Al . The handheld cableless interaction unit 
described there is used to generate and transmit 

20 location, position and/or movement data (i.e. spatial 
position coordinates of the interaction unit) for the 
purpose of three-dimensional virtual navigation in said 
scene and in any functional elements of the user 
interface and for the purpose of manipulating virtual 

25 objects in the scene. To this end, the interaction unit 
has a sensor which interacts, via a radio connection, 
with a position detection sensor system provided in the 
VR graphics system. Said position data comprise the six 
possible degrees of freedom of translation and rotation 

3 0 of the interaction unit and are evaluated in real time 
in a computer-aided manner in order to determine a 
movement or spatial trajectory of the interaction unit. 

User-guided interactions may, in principle, be 
3 5 subdivided into a logical part and a physical part. The 
logical part is the virtual three-dimensional user 
interface and includes, for example, the display of 
functions or menus, the method of selecting objects or 
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function modes and the type of navigation. The physical 
part corresponds to the equipment-related 

implementation such as the technical configuration of 
the interaction unit and the projection technology used 
5 to display the scene. 

As regards the use of said interaction units, it is 
desirable for said interactions, in particular more 
complex interactions such as function selection or menu 
10 control, to be as technically simple as possible and 
nevertheless to be capable of being controlled in a 
manner which is as safe as possible to use and is as 
operationally reliable as possible. 

15 The invention therefore proposes a method and an 
apparatus for controlling a virtual reality (VR) 
graphics system (which is concerned in this case) using 
said interactions, which method and apparatus are based 
on the inventive concept of first of all forming a 

2 0 reference system, which is arranged on the spatial or 
movement trajectory of the interaction unit, and 
evaluating subsequent interactions using this reference 
system. 

2 5 The special feature of the inventive method therefore 

resides in the fact that, as a result of a first 
interaction by the user, an initial spatial point which 
is initially fixed is determined, preferably together 
with an associated reference coordinate system, on the 

3 0 spatial trajectory of the interaction unit, and that 

the interaction unit is used to evaluate at least one 
subsequent interaction relative to the initial spatial 
point determined and the associated reference 
coordinate system. 

35 

Another refinement provides for the initial spatial 
point to represent the zero point or origin of said 
reference coordinate system and for reference or 
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threshold values to be prescribed in this coordinate 
system, a particular function or a particular menu 
selection associated with the virtual user interface, 
which has been inserted into the current scene, being 
5 effected when said reference or threshold values are 
exceeded by the instantaneous spatial position or 
spatial orientation of the interaction unit. These 
reference values are preferably located on the surface 
of a geometric body which is arranged symmetrically 

10 (imaginary) with respect to the initial spatial point, 
for example on the surface of a sphere, the surface of 
an ellipsoid, the surface of a cube, the surface of a 
cuboid, the surface of a tetrahedron or the like. The 
reference points may also be weighted in particular 

15 spatial directions in order to assign different 
sensitivities to particular functions or menu selection 
items, during three-dimensional interaction, along the 
real spatial trajectory of the interaction unit, as a 
result of which incorrect operation or incorrect inputs 

2 0 by a user are avoided even more effectively. 

Another refinement provides at least one further 
threshold value whose magnitude is greater than said at 
least one reference value, the reference coordinate 
25 system and the initial spatial point being caused to 
move to the new spatial position when said further 
threshold value is exceeded by the instantaneous 
spatial position of the interaction unit. This has the 
advantage that said advantageous method of operation of 

3 0 the reference coordinate system during said function or 

menu selection remains even in the case of 
(inadvertently) excessive changes in the position of 
the interaction unit. 

3 5 The procedure proposed according to the invention and 
the user interface which is likewise proposed afford 
the advantage, in particular, that even complex 
interactions, for example over a plurality of function 
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or menu levels, can be effected very intuitively, to be 
precise solely by means of spatial movement of the 
interaction unit. Only the determination of the first 
initial spatial point must be effected by means of a 
5 special interaction, preferably by means of a control 
element which is arranged on the interaction unit, for 
example a pushbutton or the like. In addition, control 
of the user interface by continuously evaluating said 
trajectory of the interaction unit becomes easier to 
10 handle and even more operationally reliable in 
comparison with the interaction systems which are known 
in the prior art. 

Control of the VR graphics system using the interaction 
15 unit and a user interface that is visually inserted 
into the respective scene is preferably effected either 
via a function selection that is displayed in a three- 
dimensional visual manner or via a menu system such as 
the spherical menu described, for example, in DE 101 3 2 
20 243 Al. 

The invention can be used, with said advantages, in 
cableless and cable-bound interaction units which are 
preferably hand-guided by the user. It should be 

2 5 emphasized that, in addition to said use of the 

interaction unit including said control element 
(pushbutton) , the possible interactions may also take 
place by means of acoustic or optical interactions, for 
example by means of voice, gestures or the like. In 

3 0 this case, use may be made of the input methods 

described in detail in the dissertation by A. RoSler 
entitled " Ein System fur die Entwicklung von raumlichen 
Benutzungsschnittstellen" [A system for developing 
three-dimensional user interfaces] , University of 
35 Stuttgart, published by Jost Jetter Verlag, Heimsheim, 
particularly on pages 72 ff. (chapters 4.3.2 ff.) 
thereof. In addition to the use of said interaction 
unit, the interaction modes described there such as 
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direct and indirect and absolute and relative input may 
thus be additionally used to enable, for example, 
event-oriented interpretation of movements of the 
interaction unit or a part of the user's body. 

5 

In the case of said interpretation of the user's 
gestures, it is also possible to distinguish between 
static and dynamic gestures, the temporal sequence of a 
movement being analyzed in the case of dynamic gestures 

10 and a relative position or orientation between 
individual parts of the user's body, for example, being 
analyzed in the case of static gestures. In addition, 
it is possible to distinguish between simple input 
events and interpreted and combined input events, 

15 simple input events being triggered by discrete actions 
by the user, for example the operation of said 
pushbutton, whereas interpreted events are dynamically 
interpreted, for example taking into consideration a 
time measurement, for example when a button is pressed 

20 twice ("double click"). These two input modes may 
finally be combined in any desired manner, for example 
pressing a button once with a hand, head or facial 
gesture . 

2 5 The inventive method and the apparatus are described 

below with reference to exemplary embodiments which are 
illustrated in the drawing and which reveal further 
features and advantages of the invention. In said 
exemplary embodiments, identical or functionally 

3 0 identical features are referenced using corresponding 

reference symbols. 

In the drawing: 



35 



fig. 1 



shows a simplified overview of an 
immersive VR (virtual reality) graphics 
system which is concerned in this case; 
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figs. 2a-c show spatial trajectories, which 

typically result when an interaction 
unit as shown in fig. 1 is physically 
moved in a three-dimensional manner, in 
5 order to illustrate the inventive 

procedure when evaluating these 
trajectories; and 

uses a flowchart to show the 
illustration of an inventive routine for 
controlling an interaction unit which is 
concerned in this case. 

The VR graphics system which is diagrammatically 
illustrated in fig. 1 has a projection screen 100 in 
front of which a person (user) 105 stands in order to 
view the scene 115, which is generated there via a 
projector 110, using stereoscopic glasses 120. It goes 
without saying that auto-stereoscopic screens or the 
like may also be used in the present case instead of 
the stereoscopic glasses 120. In addition, the 
projection screen 100, the projector 110 and the 
glasses 12 0 may be replaced in the present case with a 
data helmet which is known per se and then comprises 
all three functions. 

The user 105 holds an interaction unit 125 in his hand 
in order to generate preferably absolute position data 
such as the spatial position and orientation of the 
3 0 interaction unit in the physical space and to transmit 
said data to a position detection sensor system 13 0 - 
140. Alternatively, however, relative or differential 
position data may also be used but this is not 
important in the present context . 

35 

The interaction unit 125 comprises a position detection 
system 145, preferably an arrangement of optical 
measurement systems 145, both the absolute values of 



fig. 3 

10 
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the three possible angles of rotation and the absolute 
values of the translational movements of the 
interaction unit 12 5, which are possible in the three 
possible spatial directions, being detected using said 
5 arrangement of measurement systems and being processed 
in real time by a digital computer 150 in the manner 
described below. Alternatively, these position data may 
be detected using acceleration sensors, gyroscopes or 
the like which then generally provide only relative or 
10 differential position data. Since this sensor system is 
not important in the present case, a more detailed 
description is dispensed with here and reference is 
made to the documents mentioned at the outset. 

Said absolute position data are generated by a computer 
system which is connected to the interaction unit 125. 
To this end, they are transmitted to a microprocessor 
160 of a digital computer 150 in which, inter alia, the 
necessary graphical evaluation processes (which are to 
be assumed to be familiar to a person skilled in the 
art) are carried out in order to generate the 
stereoscopic three-dimensional scene 115. The three- 
dimensional scene representation 115 is used, in 
particular, for visualizing object manipulations, for 
three-dimensional navigation in the entire scene and 
for displaying function selection structures and/or 
menu structures . 

In the present exemplary embodiment, the interaction 
30 unit 125 is connected, for carrying data, to the 
digital computer 150, via a radio connection 170, using 
a reception part 165 (which is arranged there) . The 
position data which are transmitted from the sensors 
145 to the position detection sensor system 130 - 140 
3 5 are likewise transmitted in a wireless manner by radio 
links 175 - 185. 



20 
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Additionally depicted are the head position (HP) of the 
user 105 and his viewing direction (VD) 190 with 
respect to the projection screen 100 and the scene 115 
projected there. These two variables are important for 
5 calculating a current stereoscopic projection insofar 
as they considerably concomitantly determine the 
necessar y scene perspective since the perspective also 
depends, in a manner known per se, on these two 
variables . 

10 

In the present exemplary embodiment, the interaction 
unit 125 comprises a pushbutton 195 which the user 105 
can use, in addition to said possibilities for moving 
the interaction unit 12 5 in the space, to mechanically 
15 trigger an interaction, as described below with 
reference to fig. 3. It goes without saying that two or 
more pushbuttons may also alternatively be arranged in 
order to enable different interactions, if appropriate. 

2 0 The central element of the immersive VR graphics system 

shown is the stereoscopic representation (which is 
guided (tracked) using the position detection sensor 
system 130 - 140) of the respective three-dimensional 
scene data 115. In this case, the perspective of the 
25 scene representation depends on the observer's vantage 
point and on the head position (HP) and viewing 
direction (VD) . To this end, the head position (HP) is 
continuously measured using a three-dimensional 
position measurement system (not illustrated here) and 

3 0 the geometry of the view volumes for both eyes is 

adapted according to these position values. This 
position measurement system comprises a similar sensor 
system to said position detection system 130 - 140 and 
may be integrated in the latter, if appropriate. A 
3 5 separate image from the respective perspective is 
calculated for each eye. The difference (disparity) 
gives rise to the stereoscopic perception of depth. 
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In the present case, an interaction by a user is 
understood as meaning any action by the user, 
preferably using said interaction unit 125. Included in 
this case are the movement of the interaction unit 12 5 
5 on a spatial trajectory shown in figures 2a - 2c and 
the operation of one or more pushbuttons 19 5 which are 
arranged on the interaction unit 125. Acoustic actions 
by the user, for example a voice input, or an action 
determined by gestures may additionally be included. 

10 

Figures 2a - 2c then illustrate typical spatial 
trajectories 2 00 which result when the above-described 
interaction unit 12 5 is moved. It should be emphasized 
that, for the purposes of simplification, figures 2a - 

15 2c show only a two-dimensional section of the formation 
which is actually three-dimensional. In this case, 
spherical shells which are to be represented have been 
degenerated to lines, for example. The respective 
direction of movement along the course of the 

20 trajectory is indicated by arrows 203. It shall be 
assumed that the user 105 (not shown in this 
illustration) respectively uses the pushbutton 195, for 
example, at the point 205 and at the points 205', 
205' ' , 205' 1 ' on the trajectory, to signal to the VR 

2 5 graphics system (fig. 1) that an initial spatial point 
(ISP) 2 05 with an associated reference coordinate 
system 210 is to be determined. The spatial coordinates 
which correspond to the ISP and, as described above, 
are determined using the position detection sensor 

30 system 130 - 140 are transmitted to the digital 
computer 150 by radio in this case. From this time on, 
the further points on the trajectory 2 00 are calculated 
in relation to this ISP, virtually in new relative 
coordinates . 

35 

At the same time as the reference coordinate system 210 
is determined, two shells which are arranged around the 
ISP 2 05 are calculated, to be precise an inner shell 
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215 having corresponding shell segments 217 and a 
continuous (i.e. not subdivided into such shell 
segments) outer shell 22 0. It should be emphasized 
that, in the technical sense, the shells shown 
5 represent only auxiliary means when calculating said 
threshold values and when calculating or detecting when 
these threshold values have been exceeded by the 
spatial trajectory of the interaction unit 125 and 
these shells therefore do not visually appear in the 
10 scene. The inner shell 215 defines the first threshold 
value mentioned at the outset, whereas the outer shell 
represents said second threshold value. 

When penetrated by the trajectory 200, said shell 
15 segments 217 of the inner shell 215 are used to 
automatically trigger actions, preferably in a menu 
system of a user interface that is visualized in the 
present scene, to be precise actions such as opening a 
new menu item or selecting a function or a function 
2 0 mode from a multiplicity of functions or function modes 
offered. All known and conceivable manifestations, for 
example sphere-based or ellipsoid-based menus, cube- 
based or cuboid-based menus or flat transparent text 
menus, are suitable, in principle, as the menu system. 

2 5 The precise method of operation of such menu systems 

for selecting function modes or the like is described 
in detail in the two documents mentioned at the outset 
and these documents are therefore referred to in full 
in this respect in the present context . 

30 

The course of the trajectory shown in fig. 2a 
corresponds to a scenario in which the user operates 
the pushbutton 195 while moving the interaction unit 
12 5 in order to select a menu. In this case, operation 

3 5 of the pushbutton also causes a menu system to be 

visually inserted into the scene 115. The precise 
position of this insertion on the projection screen 100 
is determined on the basis of the viewing direction 
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(VD) 190 and/or the head position (HP) of the user 105. 
In this case, provision may be made for the viewing 
direction (VD) and/or the head position (HP) to be 
detected continuously or occasionally and for the 
5 precise position at which the menu system or the 
function selection system is inserted to be determined 
on the basis of the viewing direction (VD) detected 
and/or the head position (HP) detected. 

10 For reasons of symmetry (spherical symmetry of the 
above-described shells) , the present exemplary 
embodiment is likewise preferably a spherical 
symmetrical menu system, for example a spherical menu. 
It goes without saying that the spherical shells shown 

15 in figures 2a - 2c are only exemplary and may also be 
formed by cube-shaped, cuboidal or ellipsoidal shells. 
Cubic symmetrical shell shapes, for example, are thus 
suitable in menu systems which are likewise cubic 
symmetrical (cube-shaped, cuboidal or in the form of 

2 0 text) . 

The trajectory 2 00 shown in fig. 2a penetrates the 
inner shell 215 for the first time in the region of a 
first spherical shell segment 218. This triggers a 
25 first menu or function selection. The trajectory 200 
then enters the inner region of the shell 215 again at 
the level of this segment 218 in order to penetrate 
said shell 215 again at the level of a second spherical 
shell segment 222 and thus trigger a further function 

3 0 selection. The further course (indicated by dots) of 

the trajectory 2 00 is no longer important in the 
present case. 

It goes without saying that, in the simplest 
3 5 refinement, the threshold value areas shown in figures 
2a - 2c may also be formed by scalar threshold values, 
for example in the case of a cubic reference coordinate 
system instead of the spherical coordinate system shown 
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here, only a single scalar threshold value having to be 
determined in each of the three spatial directions. 

Fig. 2b illustrates the course of a trajectory in 
5 which, after the ISP 2 05 has been determined, the 
trajectory 200 first of all penetrates the inner shell 
215 in the region of a spherical shell segment 219, as 
a result of which a function selection or the like is 
again triggered. In contrast to fig. 2a, the trajectory 

10 then also penetrates the outer shell, to be precise at 
the point 22 5 shown. This penetration of the outer 
shell gives rise to the automatic correction (already 
mentioned) of the reference coordinate system 210, the 
ISP 2 05 being shifted to said penetration point, i.e. 

15 to the point 225 in the present case, in the present 
exemplary embodiment . 

In an alternative refinement, the ISP follows the 
trajectory incrementally (i.e. in incremental steps or 

2 0 virtually gradually) , either the outer shell being 
degenerated to a shell with a smaller diameter than the 
inner shell or the ISP respectively following the 
continuing trajectory incrementally as of said 
penetration point 225. As already said, the outer shell 

2 5 does not have any segmentation since it is not intended 
to trigger any use-specific events but merely said 
correction of the entire reference coordinate system 
210 . 

30 Fig. 2c is finally intended to be used to illustrate 
what happens when the interaction unit 12 5 is moved 
over a relatively long distance, after an ISP 205 has 
already been determined, for example owing to the fact 
that the user is moving over a relatively long distance 

35 in front of the projection screen 100 after a menu 
system has already been activated. As follows from the 
above description, repeatedly leaving the outer shell 
22 0 gives rise to repeated correction (three times in 
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the present exemplary embodiment) of the ISP 2 05, 
respectively resulting in ISPs 205' , 205' ' and 205' ■', 
and of the reference coordinate system 210 associated 
with said ISP, respectively resulting in reference 
5 coordinate systems 210', 210' ' and 210 ,,< which have 
been correspondingly shifted and are illustrated using 
dashed lines in the present case. 

It should be noted that, for the purpose of 
generalization, the physical spatial trajectory shown 
in figures 2a - 2c may be represented either by a pure 
translational movement or a pure rotational movement of 
the interaction unit 125 or else by a combination of 
these two types of movement. In the case of such 
rotational movements of the interaction unit in order 
to trigger particular interactions with an above- 
described menu system or in order to manipulate virtual 
objects in the scene 115, provision may additionally be 
made for the interaction to be triggered only when at 
least one second interaction, in particular using the 
control element, has been triggered. This 
advantageously prevents even a slight rotation (which 
may be undesirable) of the interaction unit 125 
triggering an interaction. It is also possible for 
rotations of the interaction unit 125 to be canceled 
again without even having to trigger an interaction in 
the VR graphics system. 

Fig. 3 now shows an exemplary embodiment of an 
3 0 inventive routine for evaluating the spatial course of 
a trajectory which has been assumed as in figures 2a - 
2c. After the start 300 of the routine, which is 
preferably triggered by switching on the VR graphics 
system or by, for instance, subsequently activating the 
35 interaction unit 125, the routine is first of all in a 
waiting loop in which a check is continuously or 
occasionally carried out 305 in order to determine 
whether the user has carried out an interaction in 
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order to determine, if appropriate, an above-described 
initial spatial point ISP 205 at the instantaneous 
spatial position of the interaction unit. This 
"initial" interaction is preferably effected using the 
5 above-described pushbutton 195 but may also be effected 
in the manner described at the outset by means of 
voice, gestures or the like. 

If such an initial interaction is determined, said 
10 reference coordinate system 210 is first of all 
determined in step 310, the coordinate origin being 
formed by the ISP 205. In subsequent steps 315 and 320, 
the reference points or reference area segments 217 of 
said first threshold 215 and the second threshold area 
15 22 0 are determined in the reference coordinate system 
210. 

Said steps are again followed by a loop in which the 
current position of the interaction unit 125 is first 
20 of all detected 325. A check is then carried out 330 in 
order to determine whether the detected value of the 
current position is outside said first threshold value 
or the value of the present reference area segment 217. 
If this condition 330 is not satisfied, the routine 

2 5 jumps back to step 32 5 in order to detect a new current 

position value of the interaction unit 125. However, if 
the condition 330 is satisfied, the trajectory has 
penetrated the first threshold area 215. In this case, 
a check is also first of all carried out 335 in order 

3 0 to determine which reference point or which reference 

area segment in the reference coordinate system is 
affected thereby. The corresponding function or menu 
selection is then triggered 340 on the basis of the 
result of the last check 335. 

35 

In the special case of the triggered function being a 
function that ends the entire routine, which is 
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additionally checked in step 342, the process jumps to 
step 343 in which the routine is then ended. 

In the case of the trajectory actually having 
5 penetrated the first threshold area 215, a check is 
also carried out 34 5 in order to determine whether the 
trajectory has also already penetrated the second 
threshold area 220. That is to say a check is also 
carried out 345 in this case in order to determine 

10 whether the magnitude of the value of the current 
position of the interaction unit 125 in the present 
reference coordinates exceeds the value of the second 
threshold. If this condition is not satisfied, the 
process jumps back to step 325 again and a new position 

15 value of the interaction unit 125 is detected. 
Otherwise, the reference coordinate system and its 
origin 350 which coincides with the ISP 2 05 are 
corrected and, if appropriate, incrementally shifted to 
the current position of the trajectory of the 

2 0 interaction unit 125. 

It should finally be noted that the above-described 
concept of the initial spatial point (ISP) also 
includes, in principle, user interfaces in which the 

25 interaction is effected using a pure rotation of the 
interaction unit 12 5 or a combination of a pure 
translation and a pure rotation. In the case of a 
rotation, the ISP can be understood as meaning an 
initial spatial angle cp = 0° of an imaginary spherical 

30 or cylindrical coordinate system. In this case, the two 
threshold values described may be formed by discrete 
angles, for example cp = 90° and cp = 180°, said 
coordinate system then being corrected, for example, by 
the angle cp = 180° when the threshold values are 

35 exceeded by cp = 180°. 



